A mobile artificial neural network apparatus

ABSTRACT

A mobile artificial neural network device is provided. The mobile artificial neural network device includes a camera configured to output a video of a product at a first frame rate, an AI recognition model configured to recognize a product information by receiving the product video, an artificial neural network processor configured to drive the AI recognition model at a second frame rate, and a display module configured to display the video of the product at the first frame rate and display the product information at the second frame rate.

BACKGROUND OF THE DISCLOSURE Technical Field

The present disclosure relates to a mobile artificial neural networkdevice, and more particularly, to a mobile artificial neural networkdevice capable of providing product information recognized using an AIrecognition model to a user.

Background Art

Generally, in the operation of a conventional online product purchasesystem, first, a consumer accesses an online shopping mall serverthrough the Internet using a browser installed in a terminal in order topurchase a product. Then, the shopping mall server transmits webpageinformation containing information on various products to thecorresponding terminal through the Internet, and the terminal displaysthe corresponding information on a display module. At this time, while auser of the terminal searches the webpage of the shopping mall serverdisplayed on the display module, various text information or photoinformation on products provided by the shopping mall server is checked.If there is a desired product, the user can select the product andpurchase it. The shopping mall server receives payment through anelectronic payment method and delivers the paid product offline.

However, in the conventional online product purchase system, it iscumbersome and inconvenient to purchase a product that a consumer wants,because in order to purchase a product that a consumer wants, it isnecessary to find the product that the consumer wants through a productsearch performed after accessing the Internet online and to graspinformation about the product. In addition, when there is a desiredproduct offline, there is a problem that it is relatively cumbersome tosearch for a product's price and information than when online.

SUMMARY OF THE DISCLOSURE

The inventor of the present disclosure has conducted research anddevelopment on a mobile terminal capable of quickly recognizinginformation on a sale product during offline shopping using anartificial neural network.

First, the inventor of the present disclosure attempted to implementaugmented reality in a mobile terminal by transmitting the videocaptured by the mobile terminal to the AI recognition model stored inthe Internet server in real time and transmitting the productinformation recognized by the AI recognition model of the Internetserver back to the mobile terminal.

However, in this method, since high-definition video must be transmittedto the Internet server in real time, the amount of data transmission issignificantly larger than that of photo information, and the AIrecognition model stored in the Internet server must sequentiallyprocess many and unspecified recognition requests. In relation to this,the inventor has recognized that it is difficult in practice for usersto monopolize the AI recognition model of the server in real time andthat the response speed can be significantly delayed depending on thenumber of users connected to the server.

Accordingly, the inventor of the present disclosure has recognized thatit is necessary to perform artificial neural network operations in amobile terminal.

Accordingly, the problem to be solved by the present disclosure is toprovide a mobile artificial neural network device, equipped with acamera in a mobile artificial neural network device, which is a mobileterminal capable of artificial neural network operation, whichrecognizes product information in real time using an AI recognitionmodel while filming a product in real time with a camera, and which isimplemented with augmented reality capable of displaying recognizedproduct information and product video on a display module at the sametime in real time.

On the other hand, the inventor of the present disclosure alsorecognized that the recognition rate (%) of the product may decreasewhen recognizing a new product with the AI recognition model that hasbeen learned and stored in the mobile artificial neural network device.That is, an AI recognition model that has not learned a new product mayrecognize it as a similar product that has already been learned, but maynot recognize the new product. Accordingly, the inventor of the presentdisclosure also recognized that the AI recognition model needs to benewly trained in order to improve the recognition rate (%) of eachproduct when a new product is released. Accordingly, the inventor of thepresent disclosure performed research on a mobile artificial neuralnetwork device capable of updating an AI recognition model in order toimprove the recognition rate (%) of newly released products. However,the inventor of the present disclosure also recognized that, for therecognition of newly released products, it is not easy for the AIrecognition model stored in the mobile artificial neural network deviceto learn by itself. More specifically, it was recognized that it maytake hours or days to learn the AI recognition model, that it isdifficult for users to directly generate new learning data for learningof newly released products, and that considerable power consumption andcomputational amount are required for learning.

Accordingly, another problem to be solved by the present disclosure isto provide a mobile artificial neural network device capable ofimproving the recognition rate (%) of newly launched products, byupdating the AI recognition model stored in the mobile artificial neuralnetwork device to the newly trained AI recognition model and minimizingthe self-learning of the mobile artificial neural network device.

Meanwhile, the inventor of the present disclosure recognized that thenumber of products that the AI recognition model can recognize can bedetermined by the product image of the training set for learning the AIrecognition model and the information label of the product. It wasfurther recognized that the big data operators that manufacture or sellproducts have the advantage of creating a training set of productsrelated to their business area.

Accordingly, another problem to be solved by the present disclosure isto provide a mobile artificial neural network device capable ofrecognizing various products by storing a plurality of different AIrecognition models learned to recognize different products.

Furthermore, the inventor of the present disclosure has also recognizedthat the recognition rate (%) of the product in the AI recognition modelof the mobile artificial neural network device can be improved when theAI recognition model of the mobile artificial neural network device islearned to recognize the unique information of the product, for example,the shape, color, trademark, name, manufacturer, and barcode of theproduct.

Accordingly, another problem to be solved by the present disclosure isto provide a mobile artificial neural network device capable ofimproving the recognition rate (%) of a product by providing an AIrecognition model that has been learned to recognize unique informationof a product.

The inventor of the present disclosure has recognized that specificinformation among the unique information of a product can be updated inreal time. For example, the sales price of a product, information ononline and offline vendors, and inventory information of a product maybe changed in real time. In other words, it was recognized that it isinefficient to learn additional information of a product that changes inreal time with an AI recognition model.

That is, since the above-described additional information on the productis very important to the user when purchasing the product, it wasrecognized that the additional information on the product is required.

In addition, it was recognized that it is efficient to classify specificproduct information as additional product information and obtain itseparately through a server.

Accordingly, another problem to be solved by the present disclosure isto provide a mobile artificial neural network device capable ofreceiving additional product information that can be updated in realtime by transmitting the product information recognized in the AIrecognition model to the server. In addition, another task is to providea mobile artificial neural network device that can assist in areasonable online or offline purchase by using the product informationrecognized by the AI recognition model and additional information of theproduct searched from the server.

On the other hand, the inventor of the present disclosure recognized theneed for an AI recognition model capable of minimizing the reduction inproduct recognition rate (%) while reducing the computational amount orpower consumption of the artificial neural network processor thatcalculates the AI recognition model to improve performance such asreducing heat generation of the mobile artificial neural network deviceand improving battery operation time.

Accordingly, another problem to be solved by the present disclosure isto provide a mobile artificial neural network device including an AIrecognition model capable of minimizing a decrease in the recognitionrate (%) of a product while reducing the computational amount or powerconsumption of the artificial neural network processor.

Accordingly, another problem to be solved by the present disclosure isto provide a mobile artificial neural network device including aprocessor capable of efficiently calculating a quantized AI recognitionmodel and a quantized AI recognition model for stable augmented realityimplementation of a mobile artificial neural network device.

The problems of the present disclosure are not limited to the problemsmentioned above, and other problems that are not mentioned will beclearly understood by those skilled in the art from the followingdescription.

In order to solve the above-described problems, a mobile artificialneural network device according to an embodiment of the presentdisclosure is provided. The mobile artificial neural network device mayinclude a camera configured to output a video of a product at a firstframe rate; an artificial intelligence (AI) recognition model configuredto recognize product information by receiving the video of the product;an artificial neural network processor configured to drive an AIrecognition model at a second frame rate; and a display moduleconfigured to display a video of a product at a first frame rate and todisplay product information at a second frame rate.

According to another feature of the present disclosure, the first framerate and the second frame rate may be the same.

According to another feature of the present disclosure, the first framerate may be faster than the second frame rate.

According to another feature of the present disclosure, the mobileartificial neural network device may further comprise a battery, and thecamera or artificial neural network processor may be configured to lowerthe first frame rate when a remaining charge of the battery falls belowthe first threshold value.

According to another feature of the present disclosure, the first framerate may be configured to be selectively adjusted in consideration ofpower consumption of the mobile artificial neural network device.

According to another feature of the present disclosure, the artificialneural network processor may be configured to include an operationstructure capable of performing an artificial neural network operationof the AI recognition model.

According to another feature of the present disclosure, productinformation may be superimposed on the video of the product to displayaugmented reality in the display module.

According to another feature of the present disclosure, mobileartificial neural network device may further comprise a communicationmodule, and the communication module may be configured to transmitinformation on the product to the server and to receive additionalinformation on the product searched from the server.

According to another feature of the present disclosure, the mobileartificial neural network device may be configured to transmit onlyproduct information among the product video and the product informationto the server through the communication module.

According to another feature of the present disclosure, the mobileartificial neural network device may be configured to transmit productinformation to the server and to receive additional product informationfrom the server.

According to another feature of the present disclosure, the AIrecognition model may be configured to recognize consecutive images of avideo of a product input from various angles, and when information ofdifferent products among product information is recognized, theinformation of different products may be combined.

According to another feature of the present disclosure, the accumulatedinformation is at least one of a shape, a color, a trademark, a name, amanufacturer, and a barcode of the product.

According to another feature of the present disclosure, the AIrecognition model is configured to recognize the video of the productand to output information of at least one product in the order of a highrecognition rate.

According to another feature of the present disclosure, the AIrecognition model is configured to be updated with the newly trained AIrecognition model through the server.

According to another feature of the present disclosure, the AIrecognition model is configured to further include a plurality ofmutually different AI recognition models.

According to another feature of the present disclosure, the AIrecognition model is configured to recognize the GS1 standard productidentification code or barcode and to receive additional information ofthe product corresponding to the GS1 standard product identificationcode or barcode through the server.

According to another feature of the present disclosure, the additionalinformation on the product includes information on the lowest pricecorresponding to the information on the product.

According to another feature of the present disclosure, the AIrecognition model is characterized in that it is a lightened AIrecognition model.

According to another feature of the present disclosure, the lightened AIrecognition model is characterized in that at least one lighteningtechniques among pruning, quantization, model compression, knowledgedistillation, and retraining, and AI-based lightening model optimizationtechniques is applied.

According to another feature of the present disclosure, the processor isan artificial neural network processor, which is an NPU.

According to a mobile artificial neural network device according tovarious embodiments of the present disclosure, there is an effect ofproviding a mobile artificial neural network device implementingaugmented reality with a camera equipped in a mobile artificial neuralnetwork device, which is a mobile terminal capable of artificial neuralnetwork operation, while shooting a product in real time with a camera.The AI recognition model is used to recognize product information inreal time, and a display module displays the recognized productinformation and the product video at the same time.

According to a mobile artificial neural network device according tovarious embodiments of the present disclosure, since the artificialneural network processor drives the AI recognition model stored in themobile artificial neural network device, the AI recognition model storedin the Internet server may not be used, and thus, there is an effect ofperforming product recognition in real time.

According to a mobile artificial neural network device according tovarious embodiments of the present disclosure, there is an effect ofimproving the recognition rate (%) of newly launched products byupdating the AI recognition model stored in the mobile artificial neuralnetwork device with the AI recognition model newly trained from theoutside, and there is an effect of removing or minimizing self-learningof the AI recognition model stored in the mobile artificial neuralnetwork device.

According to a mobile artificial neural network device according tovarious embodiments of the present disclosure, it is possible to receiveadditional information of a product that can be updated in real time bytransmitting information on a product recognized in an AI recognitionmodel to a server.

A mobile artificial neural network device according to variousembodiments of the present disclosure has an effect of further improvinga product recognition rate (%) by using product information andadditional product information.

A mobile artificial neural network device according to variousembodiments of the present disclosure has an effect of assisting areasonable online or offline purchase by using information on a productrecognized by an AI recognition model and additional information on aproduct searched from a server.

Provided are a mobile artificial neural network device according tovarious embodiments of the present disclosure, and an AI recognitionmodel that can minimize the decrease in the recognition rate (%) of aproduct while reducing an amount of calculation or power consumption ofan artificial neural network processor. Thus, there is an effect ofreducing the amount of calculation and power consumption, whileminimizing the decrease in the recognition rate of products.

The effects according to the present disclosure are not limited by thecontents exemplified above, and various additional effects are includedin the present specification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a mobile artificial neuralnetwork device configured to recognize product information according toembodiments of the present disclosure.

FIG. 2 is a schematic diagram for explaining an AI recognition modelincluded in a mobile artificial neural network device according toembodiments of the present disclosure.

FIGS. 3A and 3B are diagrams for explaining several examples ofmachine-learning methods for obtaining the learned AI recognition modelshown in FIG. 2 .

FIG. 4A is a diagram illustrating an example of a training set.

FIG. 4B is a schematic diagram for explaining an identification code anda barcode.

FIG. 5 is a diagram illustrating a method of providing additionalinformation on a product according to a fifth embodiment of the presentdisclosure.

FIG. 6 is a flowchart of a method for providing additional informationon a product according to the fifth embodiment of the present disclosureillustrated in FIG. 5 .

FIG. 7A is a schematic diagram illustrating a difference in recognitionrates between a quantized AI recognition model and an unquantized AIrecognition model.

FIG. 7B is a schematic diagram illustrating energy consumption per unitoperation of the processor 150 according to quantization.

FIG. 7C is a schematic diagram illustrating operation efficiency andoperation speed according to the type of processor 150.

DETAILED DESCRIPTION OF THE DISCLOSURE

Advantages and features of the present disclosure, and a method ofachieving them will become apparent with reference to variousembodiments described below in detail with reference to the accompanyingdrawings. However, the present disclosure is not limited to theembodiments described below and may be implemented in various formsdifferent from each other, and the embodiments of the present disclosureare merely provided to completely inform the scope of the invention tothose of ordinary skill in the art to which the present disclosurebelongs, and the present disclosure is only defined by the scope of theclaims.

A detailed description of the present disclosure may be described withreference to the drawings for convenience of description as an exampleof a specific embodiment in which the present disclosure may bepracticed. Although elements of the various embodiments of the presentdisclosure are different from each other, a manufacturing method, anoperation method, an algorithm, a shape, a process, a structure, and acharacteristic described in a specific embodiment may be combined withor included with other embodiments. Further, the location or arrangementof individual components within each disclosed embodiment may be changedwithout departing from the spirit and scope of the present disclosure.Each feature of the various embodiments of the present disclosure may bepartially or wholly combined or combined with each other, as thoseskilled in the art can fully understand, technically, variousinterlocking and driving are possible, and each of the embodiments maybe implemented independently of each other or may be implementedtogether in a related relationship.

Since the shapes, sizes, ratios, angles, numbers, and the like disclosedin the drawings for describing the embodiments of the present disclosureare exemplary, the present disclosure refers to the drawings, but is notlimited thereto. The same reference numerals refer to the same elementsthroughout the specification. Further, in describing the presentdisclosure, if it is determined that a detailed description of a relatedknown technology may unnecessarily obscure the subject matter of thepresent disclosure, the detailed description may be omitted. In case of‘include’, ‘have’, and ‘consist of’ mentioned in the presentspecification, other elements may be added unless ‘only’ is used. In thecase of expressing the constituent elements in the singular, it includesthe case of including the plural unless specifically stated otherwise.In the case of expressing an element in the singular, it includes thecase of including the plural unless specifically stated otherwise. Ininterpreting the elements, it is interpreted as including a tolerancerange even if there is no explicit description. In the case of adescription of the positional relationship, for example, when thepositional relationship of the two elements is described as ‘on’, ‘onthe upper portion’, ‘under’, ‘next to’, or ‘adjacent to’, one or moreother elements may be located between the two elements unless‘immediately’ or ‘direct’ is used. When an element or layer is referredto as ‘on’ another element or layer, it includes all cases in whichanother layer or another element is interposed directly on or anotherlayer or another element is interposed therebetween.

FIG. 1 is a schematic diagram illustrating a mobile artificial neuralnetwork device configured to recognize product information according toembodiments of the present disclosure. FIG. 2 is a schematic diagram forexplaining an AI recognition model included in a mobile artificialneural network device according to embodiments of the presentdisclosure.

The mobile artificial neural network device 100 according to the firstembodiment of the present disclosure may communicate with the server 900through the communication network 500. However, it is not limitedthereto.

The communication network 500 may use a well-known communication networksuch as Wi-Fi, LTE, 3G, 4G, and 5G, and embodiments of the presentdisclosure are not limited to the communication network 500.

The server 900 may refer to various servers capable of communicatingwith the mobile artificial neural network device 100 and may be ashopping mall server that can search for a variety of productinformation, a server that trains an AI recognition model, a server thatdistributes an AI recognition model that learns specific products, andthe like, and embodiments of the present disclosure are not limited tothe server 900.

Hereinafter, the mobile artificial neural network device 100 accordingto the first embodiment of the present disclosure will be described indetail.

The mobile artificial neural network device 100 according to the firstembodiment of the present disclosure may be configured to include adisplay module 120, an artificial neural network processor 150, and acamera 180.

The mobile artificial neural network device 100 according to the firstembodiment of the present disclosure is illustrated as a smart phone inFIG. 1 , but is not limited thereto, and is also possible to beimplemented as an electronic device such as a smart phone, a tablet PC,a personal computer, a notebook, and the like.

The artificial neural network processor 150 of the mobile artificialneural network device 100 is configured to drive the AI recognitionmodel 155. The AI recognition model 155 refers to an artificial neuralnetwork that has been machine-learned to recognize a specific product.The AI recognition model 155 will be described later with reference toFIGS. 2 to 4 .

The artificial neural network processor 150 refers to a processorconfigured to efficiently perform an artificial neural network operationrequired by the AI recognition model 155. For example, the artificialneural network processor 150, capable of artificial neural networkoperation, may be one among a central processing unit (CPU), a graphicsprocessing Unit (GPU), a microcontroller unit (MCU), a digital signalprocessor (DSP), and a neural processing unit (NPU). In addition, theartificial neural network processor 150 may be a system-on-chip (SoC)including at least two of a CPU, a GPU, an MCU, a DSP, and an NPU.However, it is not limited thereto. The artificial neural networkprocessor 150 may be configured as a separate chip separated from anapplication processor (AP), which is a main processor and is included inthe mobile artificial neural network device 100. In this case, theartificial neural network processor 150 configured with a separatededicated chip may be implemented as an NPU. The artificial neuralnetwork processor 150 is configured to have an operation structureoptimized for artificial neural network operation. That is, it isconfigured to have a structure similar to that of an artificial neuralnetwork. Accordingly, the artificial neural network processor 150 has aneffect of providing superior performance compared to a conventionalprocessor when performing tasks such as image processing and objectrecognition. In addition, in terms of power consumption, since therelatively low power consumption characteristics are superior to that ofthe conventional processor, when the mobile artificial neural networkdevice 100 recognizes product information, there is an effect that themobile artificial neural network device 100 can perform calculationprocessing quickly with relatively low power consumption.

The artificial neural network processor 150 of the mobile artificialneural network device 100 according to the first embodiment of thepresent disclosure may recognize a product at a specific frame rate/sec.The camera 180 of the mobile artificial neural network device 100 may beconfigured to record a product at a rate of, for example, 60 Hz, 30 Hz,or 15 Hz (or sixty, thirty, or fifteen frames per second). In addition,the artificial neural network processor 150 may be configured to outputa product recognition result at a specific frame rate.

The mobile artificial neural network device 100 provides a video of aproduct from the camera 180 to the AI recognition model 155 at a firstframe rate. The artificial neural network processor 150 drives the AIrecognition model 155 at the first frame rate to recognize productinformation from the video. The display module 120 may display a videoof a product filmed by the camera 180 in real time and may displayinformation of the product recognized by the AI recognition model 155 onthe video of the product to implement augmented reality.

In other words, the mobile artificial neural network device 100 maydisplay a video captured on the display module 120 and simultaneouslyinput the video to the AI recognition model 155. The artificial neuralnetwork processor 150 may drive the AI recognition model 155 to obtaininformation on the recognized product. Product information may bedisplayed together with an image on the display module 120.

The mobile artificial neural network device 100 according to the firstembodiment of the present disclosure has an effect of realizingaugmented reality by displaying product information on the displaymodule 120 in real time while filming a product with the camera 180 inreal time. Accordingly, the user utilizes the augmented realityimplemented in the display module 120 of the mobile artificial neuralnetwork device 100, and thus, in the embodiment of the presentdisclosure, there is an effect of intuitively obtaining productinformation.

The mobile artificial neural network device 100 provides the video ofthe product from the camera 180 to the AI recognition model 155 at afirst frame rate, and the artificial neural network processor 150 mayrecognize product information from the video by driving the AIrecognition model 155 at the second frame rate. At this time, thedisplay module 120 may display a video of a product filmed by the camera180 at a first frame rate, and information on a product recognized bythe AI recognition model 155 may be displayed at a second frame rate.Accordingly, it is possible to implement augmented reality by displayingproduct information displayed at the second frame rate on the video ofthe product displayed at the first frame rate. In this case, the secondframe rate may be equal to or slower than the first frame rate.

If the camera 180 of the mobile artificial neural network device 100according to the first embodiment of the present disclosure drives thefirst frame rate, and if the artificial neural network processor 150drives the AI recognition model 155 at the second frame rate, there isan effect of reducing power consumption of the artificial neural networkprocessor 150 than when the artificial neural network processor 150operates at the first frame rate.

The mobile artificial neural network device 100 according to the firstembodiment of the present disclosure may be configured to furtherinclude a battery 130. Since the mobile artificial neural network device100 is operated by the power of the battery 130, the user can easilysearch for information on a product to be searched using the mobileartificial neural network device 100 while moving. The operating speedof the artificial neural network processor 150, that is, the frame ratemay be configured to be variable according to specific conditions.

For example, the camera 180 of the mobile artificial neural networkdevice 100 or the artificial neural network processor 150 may reduce thefirst frame rate when the remaining charge of the battery 130 fallsbelow a threshold value. The threshold value may be set as, for example,30% of the remaining amount of the battery 130, and is not limitedthereto. For example, when the first frame rate is 60 Hz, it may belowered to 30 Hz under certain conditions.

According to the above-described configuration, even if the remainingcharge of the battery 130 of the mobile artificial neural network device100 falls below a certain level or a threshold value, the operationspeed of the camera 180 or the artificial neural network processor 150is higher than the threshold value, by setting the operating speed ofthe camera 180 or the artificial neural network processor 150 to belower than when the threshold value is higher than the threshold value,there is an effect that power consumption of the mobile artificialneural network device 100 can be reduced and the operating time can beextended.

If the AI recognition model is not implemented in the mobile artificialneural network device and is implemented in the server, the amount ofdata transmission compared to the video information inevitably increasesrapidly because high-definition video must be transmitted to theInternet server in real time. The AI recognition model must sequentiallyprocess a large number of unspecified recognition requests. However, itis difficult in practice for the user to monopolize the AI recognitionmodel located on the server in real time, and there may be a problemthat the response speed is delayed or the connection is disconnecteddepending on the number of users connected to the server. To furtherexplain, it was easy to implement product recognition from image filessuch as photos when the AI recognition model is located on the server,but in practice, there is a technical difficulty implementing augmentedreality that can display product recognition results in real time fromvideo captured by a camera. However, when the artificial neural networkprocessor 150 driving the AI recognition model 150 is included in themobile artificial neural network device 100, there is an effect ofimproving the above-described problems.

If the AI recognition model recognizes product information in real timewhile receiving a real-time captured video, the recognition rate (%) canbe improved compared to recognizing product information with a photo.

For example, when a user uses the mobile artificial neural networkdevice 100 to recognize product information of a hand cream, when takinga picture, the user should shoot the product at a specific angle, andtake the focus and composition into consideration. At this time, if thefocus and composition of the photo is not good, the recognition rate (%)of the AI recognition model 155 may be decreased.

However, in the case of filming a product in real time, the user mayfilm while rotating the product, and the recognition rate (%) of the AIrecognition model 155 may be improved by providing video of the productfrom various angles. In addition, if different product informationexists on the front and back of the product, for example, if there is aproduct name on the front and a barcode on the back, in case of taking apicture, at least two pictures must be supplied to the AI recognitionmodel, and that the two pictures are related to each other must berecognized by the mobile artificial neural network device 100. However,if the user films while rotating the product around the product whileoperating the AI recognition model 155 in real time, the AI recognitionmodel 155 can determine the continuity of the images, and information onthe front and back sides is also provided sequentially, and thus, therecognition rate (%) can be improved.

That is, the AI recognition model 155 may determine the continuity ofimages of a video of products input from various angles and mayaccumulate information of different products among product information.For example, the accumulated product information may be at least one ofa shape, a color, a trademark, a name, a manufacturer, and a barcode ofthe product. Therefore, the AI recognition model 155 recognizesconsecutive images of a video of products input from various angles andrecognizes information of different products from product information.At this time, information of different products may be combined.

The mobile artificial neural network device 100 according to the firstembodiment of the present disclosure may be practiced with amodification.

For example, the mobile artificial neural network device 100 may beconfigured of a camera 180 for filming a product that the user wants tosearch for at a first frame rate, an artificial neural network processor150 configured to receive a video of the camera 180 at a first framerate and to recognize product information of the video at a first framerate through the AI recognition model 155, a communication module 170configured to transmit the recognition result to the server 900 and toreceive a search result corresponding to the transmitted recognitionresult from the server 900, and a display module 120 that displays anaugmented reality in which a search result is displayed on a video beingfilmed in real time by the camera 180.

To further explain, the mobile artificial neural network device 100 isconfigured to transmit and receive data by being connected to the server900 through a communication network 500 capable of communicating withthe communication module 170. For example, the mobile artificial neuralnetwork device 100 transmits product information recognized by the AIrecognition model 155 driven by the artificial neural network processor150 to the server 900 through the communication network 500. Inaddition, the mobile artificial neural network device 100 is configuredto receive a search result of the server 900.

To further explain, the mobile artificial neural network device 100 mayprocess product information output from the AI recognition model 155into information in the form of a query and may transmit it to theserver 900. A query is a request for information to be searched in thedatabase of the server 900. That is, the mobile artificial neuralnetwork device 100 does not transmit the captured video to the server900, but transmits only the product information output from the AIrecognition model 155 to the server 900. The communication module 170 isconfigured so that the mobile artificial neural network device 100transmits the output product information to the server 900 through thecommunication network 500. The artificial neural network processor 150may receive product information output from the AI recognition model155, may provide it to the communication module 170, and may transmitinformation on a product to the server 900 through the communicationnetwork 500 provided with the communication module 170.

According to the above configuration, modified embodiment of the mobileartificial neural network device 100 has an effect of implementing theaugmented reality by searching the server 900 for product names andprices in real time while filming a product with the camera 180 in realtime, and displaying the searched product information on the displaymodule 120. Therefore, there is an effect that the user can convenientlyand quickly search the information on the filmed product and make areasonable online or offline purchase.

Some elements of the mobile artificial neural network device 100according to the first embodiment of the present disclosure may beexcluded, and various modifications may be made. For example, the firstembodiment of the present disclosure may be modified into a fixedartificial neural network device that does not require mobility, and inthis case, the battery element may be excluded. For example, the firstembodiment of the present disclosure may be implemented with a mobileartificial neural network device excluding a camera, and in this case, aseparate camera may be connected with the mobile artificial neuralnetwork device.

Referring to FIG. 2 , the AI recognition model 155 refers to a model 15that is trained in a separate machine-learning device to perform anobject recognition function. The AI recognition model 155 may beembedded in the artificial neural network processor 150 or stored in aseparate memory of the mobile artificial neural network device 100. Whenthe AI recognition model 155 is operated, it may be implemented in amanner that is loaded into the artificial neural network processor 150.

As shown in FIG. 2 , the model 15 machine-learned with big data 300prepared in advance may be stored in the mobile artificial neuralnetwork device 100. Further, the trained model 15 may be referred to asan AI recognition model 155.

The generation of the trained model 15 may be performed in a separatemachine-learning device. In the machine-learning device, the trainedmodel 15 may be obtained by repeatedly machine-learning the big data 300on an artificial neural network prepared in advance. It will bedescribed in more detail with reference to FIGS. 3A and 3B.

FIGS. 3A and 3B are diagrams for explaining several examples ofmachine-learning methods for obtaining the learned AI recognition modelshown in FIG. 2 .

Referring to FIG. 3A, the trained model 15 shown in FIG. 2 can beobtained by machine-learning the artificial neural network by repeatedlyproviding big data 300 to the fully connected artificial neural networkas shown on the right.

As an example of an artificial neural network, the artificial neuralnetwork may include an input node (x0, x1, . . . , xi, . . . , xf−1, xf)into which an image is input, an output node (y0, y1, . . . , yi, . . ., ym−1, ym) which outputs product information of the input image, hiddennodes between the input node (x0, x1, . . . , xi, . . . , xf−1, xf) andthe output node (y0, y1, . . . , yi, . . . , ym−1, ym), and multipleassociated parameters (weight) between the output node (y0, y1, . . . ,yi, . . . , ym−1, ym) and the input node (x0, x1, . . . , xi, . . . ,xf−1, xf).

The input node (x0, x1, . . . , xi, . . . , xf−1, xf) is a nodeconfiguring an input layer and receives an image from the outside, andthe output node (y0, y1, . . . , yi, . . . , ym−1, ym) is a nodeconfiguring an output layer and outputs predetermined output data to theoutside. The hidden nodes disposed between the input node (x0, x1, . . ., xi, . . . , xf−1, xf) and the output node (y0, y1, . . . , yi, . . . ,ym−1, ym) are nodes configuring a hidden layer and connect output dataof the input node (x0, x1, . . . , xi, . . . , xf−1, xf) to input dataof the output node (y0, y1, . . . , yi, . . . , ym−1, ym). Three hiddenlayers are illustrated in FIG. 3A, but according to an embodiment, aneural network circuit may be implemented by disposing a plurality ofhidden layers, for example, two or four or more hidden layers, betweenthe input layer and the output layer.

Each input node (x0, x1, . . . , xi, . . . , xf−1, xf) of the inputlayer may be fully connected or incompletely connected to each outputnode (y0, y1, . . . , yi, . . . , ym−1, ym) of the output layer, asillustrated in FIG. 3A.

The input node (x0, x1, . . . , xi, . . . , xf−1, xf) serves to receiveinput data from the outside and deliver it to the hidden node. Then, apractical calculation is performed in the hidden node. After output datais output from the hidden node, the output node (y0, y1, . . . , yi, . .. , ym−1, ym) receives the output data and performs calculation again.When performing calculations in the hidden node and the output node (y0,y1, . . . , yi, . . . , ym−1, ym), the calculation is performed bymultiplying the input data that is input to an own node by apredetermined associated parameter (or weight, w). After resultantcalculation values performed in respective nodes are summed (weightedsum), predetermined output data is output by passing the sum through apreset activation function.

The hidden node and the output node (y0, y1, . . . , yi, . . . , ym−1,ym) have an activation function. The activation function may be oneamong a step function, a sign function, a linear function, a logisticsigmoid function, a hyper tangent function, a ReLU function, and asoftmax function. The activation function may be appropriatelydetermined by a skilled person according to a learning method of anartificial neural network.

The artificial neural network performs machine-learning by repeatedlyupdating or modifying the associated parameter (w) to an appropriatevalue. Representative methods of machine-learning by the artificialneural network include supervised learning and unsupervised learning.

Supervised learning is a learning method in which the associatedparameter (w) is updated so that output data obtained by putting theinput data into the neural network becomes close to the target data whenthere is a clearly defined target output data that is expected to becomputed by an arbitrary neural network from input data. A multilayerstructure of FIG. 3A may be generated based on supervised learning.

Referring to FIG. 3B, illustrating another example of the artificialneural network, there is a convolutional neural network (CNN), which isa type of deep neural network (DNN). A convolutional neural network(CNN) is a neural network having one or several convolutional layers, apooling layer, and a fully connected layer. The convolutional neuralnetwork (CNN) has a structure suitable for training two-dimensional dataand can be trained through a backpropagation algorithm. It is one of therepresentative models of DNN that is widely used in various applicationfields such as object classification and object detection in images.

Meanwhile, the AI recognition models according to various embodiments ofthe present disclosure may be DNN, but are not limited thereto. Forexample, AI recognition models may be based on at least one selectedalgorithm among VGG net, R, DenseNet, a fully convolutional network(FCN) having an encoder-decoder structure, SegNet, DeconvNet, DeepLABV3+, and a deep neural network (DNN) such as U-net, SqueezeNet, Alexnet,ResNet18, MobileNet-v2, GoogLeNet, Resnet-v2, Resnet50, Resnet101, andInception-v3. Furthermore, the AI recognition models may be ensemblemodels based on at least two algorithm models among the aforementionedalgorithms.

Here, it should be noted that the artificial neural network of thepresent disclosure is not limited to the artificial neural network shownin FIGS. 3A and 3B, and the trained model 15 may be obtained bymachine-learning the big data 300 in various other artificial neuralnetworks.

Referring to FIG. 2 again, the big data 300, which is prepared inadvance, includes a training set for machine-learning of the artificialneural network described above. As illustrated in FIG. 4 , the trainingset of the big data 300 includes a plurality of product images andproduct information labels of the corresponding product images. Productinformation labels (Label 1, . . . , Label 10, . . . , Label 100, . . ., Label 1000) corresponding to each of a plurality of product images(Image 1, . . . , Image 10, . . . , Image 100, . . . , Image 1000) areprepared in advance.

The product information label corresponding to each product image isconfigured to include information on at least one of product shape,color, trademark, name, manufacturer, and barcode.

The prepared training set may be provided to the artificial neuralnetwork illustrated in FIG. 3A or 3B to acquire the trained model 15illustrated in FIG. 2 . The acquired trained model 15 is mounted on themobile artificial neural network device 100 as shown in FIG. 2 to becomean AI recognition model 155.

When an image of a specific product is captured by the AI recognitionmodel 155 mounted on the mobile artificial neural network device 100according to the first embodiment of the present disclosure, the AIrecognition model 155 outputs product information corresponding to theinput image. Here, the product information may output at least one ofthe product shape, color, trademark, name, manufacturer, and barcodecorresponding to the product information label of the training set.Specifically, when an image is received, the AI recognition model 155,may output a probability value (%) for each information of a pluralityof products classified in advance, that is, a recognition rate (%), mayinfer the information of the product with the largest probability value(%) among the probability values (%) for each information of the outputproduct as information of the product corresponding to the input image,and may output inferred product information. At this time, the productinformation may be configured to output a recognition rate (%).

However, the AI recognition model 155 may output information on aproduct having the highest probability value, but is not limitedthereto, and may output information on a plurality of products in theorder of the highest probability value. In this case, the display module120 has an effect of providing an augmented reality in which a video ofa filmed product and information of at least one product aresimultaneously displayed. In addition, the user may select the desiredproduct information when information on at least one product isdisplayed.

The mobile artificial neural network device 100 according to the firstembodiment of the present disclosure has the advantage of realizingaugmented reality while filming a product in real time, but is notlimited thereto, and it may be modified to implement the same functionusing a picture taken by the camera 180.

The mobile artificial neural network device 100 according to the secondembodiment of the present disclosure is configured to update the AIrecognition model 155.

The portable artificial neural network apparatus according to the secondembodiment of the present disclosure 100 is substantially similar to themobile artificial neural network device 100 according to the firstembodiment of the present disclosure, except that the AI recognitionmodel 155 can be updated, and thus, the redundant descriptionhereinafter may be omitted only for convenience of description.

As described above, when the learning is completed and the AIrecognition model 155 stored in the mobile artificial neural networkdevice 100 recognizes a brand-new product, the recognition rate (%) ofthe product may decrease. That is, the AI recognition model 155, whichhas not learned a new product, may be recognized as a similar productthat has already been learned, but may not accurately recognize the newproduct. However, it is not easy for the AI recognition model 155 storedin the mobile artificial neural network device 100 to learn by itselffor recognition of brand-new products.

Accordingly, the mobile artificial neural network device 100 accordingto the second embodiment of the present disclosure is configured toupdate the pre-stored AI recognition model 155.

For example, the server 900 may update the AI recognition model 155mounted on the mobile artificial neural network device 100.Specifically, the server 900 may change a parameter (weight w) and/or abias (b) of the artificial neural network of the AI recognition model155. By updating the AI recognition model 155, the recognition rate ofproduct information may be improved. The artificial neural networkprocessor 150 of the mobile artificial neural network device 100receives update information for updating the AI recognition model 155from the server 900, and thus, the AI recognition model 155 may beupdated based on the received update information, but is not limitedthereto, and the mobile artificial neural network device 100 may updatethe AI recognition model 155 through a memory device such as a CD, DVD,USB, or HDD.

For example, when recognizing a product, the mobile artificial neuralnetwork device 100 may allow a user to input an update of the AIrecognition model 155 when the recognition rate (%) is output below athreshold.

For example, the user may provide feedback to the server 900 on theproduct recognition result. Alternatively, the server 900 may transmitthe update news of the new AI recognition model 155 to the mobileartificial neural network device 100. Alternatively, the mobileartificial neural network device 100 may update the newly trained AIrecognition model 155 through the server 900 or an automatic updatefunction, but is not limited thereto, and it is possible to instruct theupdate of the AI recognition model 155 through various methods.

For example, there may be at least one server 900. To further explain, amanufacturer of a specific product may periodically or as necessaryupdate a training set in which information on newly released products isupdated. For example, a manager of the big data 300 that sells sportinggoods may update a training set in which information on products ofnewly released sporting goods is updated. In addition, the newly trainedAI recognition model can be trained with an updated training set in aseparate machine-learning device.

Accordingly, the mobile artificial neural network device 100 accordingto the second embodiment of the present disclosure uses the newlytrained AI recognition model 155, thereby improving the recognition rate(%) of newly released products.

According to the mobile artificial neural network device 100 accordingto the second embodiment of the present disclosure, there is an effectthat the previously stored AI recognition model 155 can be easilychanged as necessary to an AI recognition model obtained by learning anewly released product from the outside.

The AI recognition model 155 of the mobile artificial neural networkdevice 100 according to the third embodiment of the present disclosureis configured to store at least one AI recognition model 155.

The portable artificial neural network apparatus according to the thirdembodiment of the present disclosure 100 is substantially similar to themobile artificial neural network device 100 according to the firstembodiment of the present disclosure, except that a plurality of AIrecognition models 155 can be stored, and thus, the redundantdescription hereinafter may be omitted only for convenience ofdescription.

The mobile artificial neural network device 100 according to the thirdembodiment of the present disclosure is configured to include aplurality of AI recognition models 155. For example, the AI recognitionmodel 155 may include a first AI recognition model and a second AIrecognition model. Here, the first AI recognition model may be arecognition model that is learned to recognize as a product the sportinggoods of a sporting goods manufacturer. The second AI recognition modelmay be a recognition model learned to recognize skin care products of acosmetic product manufacturer. However, it is not limited thereto.

The plurality of AI recognition models 155 may have a specifichierarchy. For example, a product category can be stratified into alarge classification, a medium classification, and a smallclassification.

For example, the first AI recognition model may be a model trained torecognize a category of a product, which is a large category. The firstAI recognition model may be a model that has been trained to recognizeproduct categories such as beverages, toys, cosmetics, and kitchenutensils and may recognize, for example, about 1000 product categories.

For example, the second AI recognition model may be a model trained torecognize products related to a category of a specific product of alarge classification. For example, when the first AI recognition modelrecognizes beverages in a product category of a large category, themobile artificial neural network device 100 may call a second AIrecognition model that has learned to recognize beverages. In this case,the second AI recognition model may be a model trained to recognizeproduct names of about 1000 beverages.

For example, the third AI recognition model may be a smallclassification recognition model that has been trained to recognizeadditional information of a product recognized in the middleclassification. For example, it is possible to recognize whether thecontainer of the recognized beverage is a glass bottle, a can, or a PETbottle, and the capacity of the container may be recognized.

Accordingly, the mobile artificial neural network device 100 calls asecond AI recognition model corresponding to the large category productcategory recognized by the first AI recognition model, and thus, thesecond AI recognition model may recognize the product in more detail.

In addition, if necessary, the mobile artificial neural network device100 calls a third AI recognition model corresponding to the middlecategory product category recognized by the second AI recognition model,and thus, the third AI recognition model can recognize the product inmore detail.

The mobile artificial neural network device 100 according to the thirdembodiment of the present disclosure may be configured to be able tosearch and download a specific AI recognition model 155 through theserver 900 as necessary.

For example, the mobile artificial neural network device 100 may film abrand, logo, or QR code of a specific store with the camera 180, maysearch through the server 900 for the AI recognition model 155 that haslearned the corresponding product, and may store the searched AIrecognition model 155 in the mobile artificial neural network device100.

Therefore, according to the third embodiment of the present disclosure,the mobile artificial neural network device 100 has an effect ofproviding product information at a high recognition rate (%) by using aspecific AI recognition model having a high recognition rate (%) amongthe AI recognition models 155.

Therefore, since the mobile artificial neural network device 100according to the third embodiment of the present disclosure isconfigured to include a plurality of different AI recognition models,there is an effect of improving the recognition rate (%) of a specificproduct.

Therefore, as the mobile artificial neural network device 100 accordingto the third embodiment of the present disclosure is configured toinclude a plurality of hierarchical AI recognition models, there is aneffect of dramatically improving the number of recognizable productswhile improving the recognition rate (%) of a specific product.

Therefore, the mobile artificial neural network device 100 according tothe third embodiment of the present disclosure may search for a productor brand desired by the user with the AI recognition model 155, maysearch for an additional AI recognition model that has learned moredetailed product information related to the recognition result, and maystore the searched AI recognition model in the mobile artificial neuralnetwork device 100.

The mobile artificial neural network device 100 according to the fourthembodiment of the present disclosure is characterized in that the AIrecognition model 155 is configured to store an AI recognition model 155capable of recognizing a barcode or a QR code.

The mobile artificial neural network device 100 according to the fourthembodiment of the present disclosure is characterized in that it isconfigured to include an AI recognition model 155 that is learned torecognize unique identification information of a product, for example,an identification code or a barcode. Here, the AI recognition model 155may be configured to simultaneously recognize product information andbarcodes with one AI recognition model, or may be configured to furtherinclude an AI recognition model learned to recognize only separatebarcodes.

The AI recognition model 155 is characterized by being trained torecognize a GS1 standard product identification code or barcode.

The GS1 standard product identification code is a code used for productidentification and refers to a product identification code that followsan industrial, national, or internationally agreed system. The GS1standard product identification code is an international standardproduct identification code managed and distributed by GS1, which iscomposed of a network of 110 countries around the world.

FIG. 4B is a schematic diagram for explaining an identification code anda barcode. Referring to FIG. 4B, the barcode means that theidentification code is expressed in a bar shape so that the machine canread it. It is configured that by changing the thickness of the bar andthe width of the space between the bar and the bar, the identificationcode is displayed and the machine can read it optically.

The GS1 standard identification code for a single product ischaracterized by being unique worldwide. In other words, the GS1standard identification code for the same product is the same worldwide.Since only one GS1 standard identification code is assigned to oneproduct, there is an advantage that the same GS1 identification code isnot assigned to different products.

The mobile artificial neural network device 100 may recognize a barcodeor an identification code and may transmit the recognized productinformation to the server 900 in order to receive additional productinformation from the server 900. To further explain, the identificationcode does not contain information such as product name and price.Accordingly, the recognized GS1 standard identification code informationmay be obtained from the server 900 including the GS1 standardidentification code database.

According to the above configuration, the mobile artificial neuralnetwork device 100 according to the fourth embodiment of the presentdisclosure recognizes a barcode and receives a search resultcorresponding to the barcode to the server 900. Thus, there is an effectof further improving additional product information and productrecognition rate (%). In addition, even if the AI recognition model 155does not learn about a specific product, the product information may besearched from the database of the server 900 by recognizing the GS1standard identification code or barcode, and thus, there is an effect offurther improving the recognition rate (%) of the product.

The mobile artificial neural network device 100 according to the fifthembodiment of the present disclosure display the product information 160recognized by the AI recognition model 155 on the display module 120,send product information 160 to the server 900, search for additionalproduct information 165 through the server 900, and provide productinformation 160 and additional product information 165 to the user.Accordingly, the mobile artificial neural network device 100 may provideadditional information on a product that can help shopping.

FIG. 5 is a diagram illustrating a method of providing additionalinformation on a product according to a fifth embodiment of the presentdisclosure, and FIG. 6 is a flowchart of a method for providingadditional information on a product according to the fifth embodiment ofthe present disclosure illustrated in FIG. 5 .

Referring to FIGS. 5 and 6 , a camera-linked application installed inthe mobile artificial neural network device 100 is executed 601 by auser. Here, the camera-linked application (or “app”) may be a camera appbasically installed in the mobile artificial neural network device 100or may be a shopping app capable of communicating with the server 900while driving and controlling the camera 180 of the mobile artificialneural network device 100.

When the camera-linked app is executed (601) in the mobile artificialneural network device 100, a video captured by the camera 180 isdisplayed on the display module 120, and the AI recognition model 155may receive the video. When a user moves the mobile artificial neuralnetwork device 100 to film a specific product, for example, a handcream, a hand cream video may be displayed on the display module 120. Atthis time, the AI recognition model 155 attempts to recognize theproduct using the received hand cream video.

Next, the camera 180 of the mobile artificial neural network device 100provides a video to the AI recognition model 155, and the AI recognitionmodel 155 recognizes (602) the product information 160 from the video.For example, the product information 160 may include at least one of ashape, a color, a trademark, a name, a manufacturer, and a barcode ofthe product included in the input video. However, it is not limitedthereto, and various information capable of identifying the product maybe included in the product information 160.

The mobile artificial neural network device 100 may simultaneouslydisplay a video of a product and recognize the product information 160and the product video and product information 160 may be displayed onthe display module 120 in real time to implement augmented reality.

The mobile artificial neural network device 100 may display productinformation 160 to a user and may display a search window 190 on thedisplay module 120, asking whether to search for additional productinformation, for example, the lowest price information, a sales site,and the like through the server 900. Here, when a user touches thesearch window 190, that is, when input of a user is received through thesearch window 190, the mobile artificial neural network device 100transmits the recognized product information 160 to the server 900.

The artificial neural network processor 150 of the mobile artificialneural network device 100 may control to transmit (603) the productinformation 160 recognized by the AI recognition model 155 to the server900 through the communication module 170. Here, the artificial neuralnetwork processor 150 may process the product information recognized bythe AI recognition model 155 into information in the form of a query andtransmit it to the server 900.

On the other hand, when the user moves the mobile artificial neuralnetwork device 100 so that the product does not appear in the video, theproduct information 160 may not be transmitted to the server 900.

The server 900 receiving the product information 160 outputs (604)additional information corresponding to the received product information160. Here, the additional information on the product may include thelowest price information corresponding to the information on theproduct.

The server 900 transmits (605) additional information including thelowest price information to the mobile artificial neural network device100 through the communication network 500. Here, the server 900 mayprocess the additional information into information in the form of aquery and may transmit it to the mobile artificial neural network device100.

The mobile artificial neural network device 100 receives additionalproduct information including the lowest price information from theserver 900 through the communication module 170. The artificial neuralnetwork processor 150 of the mobile artificial neural network device 100outputs (607) the received additional information to the display module120 of the mobile artificial neural network device 100. Here, theartificial neural network processor 150 may display the additionalinformation of the product including the received lowest priceinformation on the display module 120 as shown in FIG. 6 in a presetmanner.

When the user selects a shopping mall desired by the user from thelowest price information displayed on the display module 120 of themobile artificial neural network device 100, the artificial neuralnetwork processor 150 may display a purchase page of a correspondingproduct of the selected shopping mall on the display module 120.

Meanwhile, the server 900 pre-stores 650 additional information onproducts including information on the lowest price for each product. Inaddition, the server 900 may update 630 and store additional productinformation including the lowest price information for each product inreal time or periodically.

In addition, the server 900 may transmit additional product informationcorresponding to product information provided from a mobile artificialneural network device 100′ or 100″, other than the mobile artificialneural network device 100, to another mobile artificial neural networkdevice 100′ or 100″.

In this manner, according to the method of providing additionalinformation of a product according to the fifth embodiment of thepresent disclosure, a user can obtain the brand and source of theproduct they want in a short time and almost in real time through theartificial neural network-based AI recognition model 155 withoutsearching through the touch screen. This is due to that a videocontaining a product is acquired through the camera 180 from thecamera-linked app installed in the mobile artificial neural networkdevice 100 and information on the product exist in the video obtainedusing the AI recognition model 155 is obtained.

In addition, according to the method for providing additionalinformation on a product using an artificial neural network according tothe fifth embodiment of the present disclosure, the captured productimage does not need to be transmitted to the server 900, and the productimage captured by the server 900 does not need to be analyzed. Thus, itis possible to provide additional information, including information onthe lowest price, sales location, and inventory to the user in realtime. This is because of that product information obtained through theAI recognition model 155, for example, product information convertedinto a query form, is transmitted to the server 900 through thecommunication network 500, and at the server 900, additional informationincluding the lowest price information corresponding to the transmittedproduct information is searched and provided to the mobile artificialneural network device 100 through the communication network 500.

According to various embodiments of the present disclosure, the mobileartificial neural network device 100 is configured to be capable ofprocessing a product of a video input from the camera 180 at a speedcapable of realizing augmented reality.

In general, the server 900 is superior to the mobile artificial neuralnetwork device 100 in terms of processing speed of a processor, hardwareresources such as memory, and processing performance. That is, since themobile artificial neural network device 100 performs an operation at theserver 900 level, the above-described technical difficulties exist.

The mobile artificial neural network device 100 according to theembodiments of the present disclosure is characterized in that it isconfigured to provide an optimized AI recognition model 155 and aprocessor 150 in consideration of hardware characteristics such as alimitedly provided battery and computing power of an arithmeticprocessing device and a storage capacity of a memory device.

That is, the AI recognition model 155 and/or the processor 150 accordingto the embodiments of the present disclosure may be configured tooperate in a portable device rather than a high-performance server orsuper computer.

Specifically, the AI recognition model 155 can be lightened. Throughlightening processes, for example, pruning, quantization, deep learningmodel parameter compression, transfer learning technology such asknowledge distillation, retraining-based lightening technology, AI-basedlightening model optimization technology, and the like can enhanceusability in portable devices. However, the present disclosure is notlimited to the above-described lightening technology, and otherlightening technologies may be used.

As a result, when the artificial neural network model is lightened, thehardware resources, computational power requirements, power consumption,heat generation, and the like required to implement the applicationservice are proportionally reduced, and the computational performanceincreases, thus maximizing the utility in portable devices.

To further explain, pruning is a technology that converts the AIrecognition model 155 to a smaller and more efficient manner as one ofthe deep learning lightening technologies. Specifically, pruningreplaces small values close to zero among associated parameters (w) orweight values of the artificial neural network with zero. The artificialneural network operation includes a vast matrix multiplication, andsince the weight value that becomes 0 makes the result of themultiplication to 0 regardless of the value of the other operand in themultiplication, in this case, the result value can be derived withoutactually executing the multiplication operation. For example, in a deeplearning model such as VGG16, even if a weight value of 90 percent ormore is substituted with 0, there is little reduction in the recognitionrate (%), so about 90 percent of the total inference calculation can beperformed without actual hardware calculation. Therefore, the pruned AIrecognition model 155 has an effect of providing an advantage suitablefor being applied to the mobile artificial neural network device 100.

To further explain, quantization is a technique for reducing the numberof bits of data. Specifically, data input to nodes of an input layer, ahidden layer, and an output layer of the AI recognition model 155 may bequantized. That is, the number of bits of data can be reduced to aspecific number of bits. For example, input data of 32-bitfloating-point can be quantized to 16-bit, 8-bit, 4-bit, or 2-bit data.In addition, the associated parameter w of the AI recognition model 155,that is, a weight value may be quantized. That is, the number of bits ofthe associated parameter calculated with the data input to each node canbe reduced to a specific number of bits. For example, a 32-bitfloating-point associated parameter can be quantized to 16-bit, 8-bit,4-bit, or 2-bit. That is, the mobile artificial neural network device100 may be configured to perform a lower-bit mathematical operation witha quantized node and an associated parameter. The quantized AIrecognition model 155 provides an advantage suitable for being appliedto the mobile artificial neural network device 100.

To further explain, deep learning parameter compression or modelcompression compresses the value of the associated parameter (w) of theAI recognition model 155 or the activation map or feature map data usingan existing data compression technique. Accordingly, the size of the AIrecognition model 155 compressed into data of a smaller size may bereduced when stored in a memory device. In particular, since the size ofthe data used in the artificial neural network is vast, the datathroughput can be reduced several times to tens of times through thiscompression, and the power consumption and delay time required formemory access can be reduced by the compressed AI recognition model 155.Therefore, the compressed AI recognition model 155 may provide anadvantage suitable for being applied to the mobile artificial neuralnetwork device 100.

To further explain, knowledge distillation is a kind of transferlearning technology, which is a technology that learns a smallartificial neural network model to be used in practice by using a largeartificial neural network model that has been well learned in the past.

For example, a large artificial neural network that has beenwell-learned in the past may be an artificial neural network consistingof about 100 of an input layer, hidden layers, and an output layer, anda small artificial neural network is an artificial neural networkconsisting of about fifty layers of an input layer, hidden layers, andan output layer.

That is, a large artificial neural network model having a relativelylarge number of layers and a relatively large weight value can implementa relatively high level of artificial intelligence. However, it isdifficult to easily implement high-level artificial intelligence in ahardware resource-constrained environment such as a portable device. Inthis case, if a lightened artificial neural network is trained usingdata and information of a large artificial neural network of highartificial intelligence that has been previously learned, a high levelof artificial intelligence can be implemented in the lightenedartificial neural network. Accordingly, the performance of the mobileartificial neural network device 100 having limited hardware resourcescan be improved.

To further explain, when applying various techniques for reducing weightof models, for example, quantization, pruning, and parametercompression, the recognition rate (%) of the AI recognition model 155may decrease. In this case, the AI recognition model 155 may beretrained. In this case, the recognition rate (%) of the AI recognitionmodel 155 may increase again. Accordingly, the performance of the mobileartificial neural network device 100 having limited hardware resourcescan be improved.

To further explain, AI-based lightened model optimization technologies,such as Neural Architecture Search or AutoML, are representative.

The AI-based lightening model optimization technology such that a methodof creating an optimally lightened artificial neural network modelthrough the process of searching the artificial neural network modelstructure through artificial intelligence such as reinforcement learningor lightening methods such as quantization, pruning, model compression,i.e., data compression, is not based on the conventional algorithms.This technology is a way for artificial intelligence to go through itsown weight reduction process to achieve optimal weight reductionresults. However, the present disclosure is not limited to theabove-described lightening technology, and other lightening technologiesmay be used.

FIG. 7A is a schematic diagram illustrating a difference between arecognition rate (%) of a quantized AI recognition model and anunquantized AI recognition model.

Referring to FIG. 7A, an AI recognition model 155 implemented as aResnet18 deep learning model that learned product recognition wasquantized from 32-bit floating-point multiplication to 4-bit integermultiplication.

The recognition rate (%) ranged from 69.758% to 69.674%, and the actualreduction in recognition rate was negligible. In other words, there wasno substantial reduction in recognition rate (%) due to quantization.The size of the weight value of the Resnet18 model before quantizationis 44.6 MB, but the size of the weight value of the quantized model isreduced to 5.5 MB. Hence, as the size of the quantized AI recognitionmodel 155 decreases, there is an effect of reducing the amount of memorystorage requirement, the delay time required for memory access, powerconsumption during memory access, the amount of hardware resource demandin the NPU, the power consumption of the NPU, and the like, and there isan effect of increasing the computational performance of the artificialneural network of the NPU.

FIG. 7B is a schematic diagram illustrating energy consumption per unitoperation of the processor 150 according to quantization.

Referring to FIG. 7B, energy consumption per unit operation of theprocessor 150 may be described as an addition operation and amultiplication operation. “8h Add” means an 8-bit integer additionoperation. “16b Add” means a 16-bit integer addition operation. “32hAdd” means a 32-bit integer addition operation. “16b FP Add” means a16-bit floating point addition operation. “32h FP Add” means a 32-bitfloating point addition operation. “4h Mult” means a 4-bit integermultiplication operation. “8b Mult” means an 8-bit integermultiplication operation. “16b Mult” means a 16-bit integermultiplication operation. “32h Mult” means a 32-bit integermultiplication operation. “16b FP Mult” refers to a 16-bitfloating-point multiplication operation. “32h FP Mult” refers to a32-bit floating-point multiplication operation. The energy unit is thepico-joule (pj).

When the processor 150 performs 32-bit floating-point multiplication and4-bit integer multiplication, the energy consumption per unit operationis approximately 37 times different. That is, when the quantized AIrecognition model 155 is calculated by the processor 150 of the mobileartificial neural network device 100, power consumption can besignificantly reduced.

FIG. 7C is a schematic diagram illustrating operation efficiency andoperation speed according to the type of processor 150.

The CPU is a general central processing unit and is a processor capableof processing various operations and efficiently performingmultitasking. However, the CPU has the advantage of being able toperform serial processing operations at high speed. The GPU is agraphics processing device, has a structure optimized for parallelprocessing, and is a processor capable of efficiently performing imageprocessing. The NPU is an artificial neural network processing deviceand has a structure optimized for matrix computation of quantized data.

Referring to FIG. 7C, the processor 150 is an artificial neural networkprocessor, which is an NPU. Since the NPU is optimized for thecomputation of the AI recognition model 155 and can be quantized, thecomputational speed can be 25 times faster than the CPU capable ofartificial neural network computation, and the computational efficiencycan be better by 50 times or more.

That is, when the quantized AI recognition model 155 is driven in theprocessor 150 in which the NPU is implemented, the performance of theembodiments of the present disclosure may be improved. However, thepresent disclosure is not limited thereto.

According to the above-described configuration, the mobile artificialneural network device 100 can recognize products at a high frame rate inlow power and low memory bandwidth.

A mobile artificial neural network device according to embodiments ofthe present disclosure may include a camera configured to output a videoof a product at a first frame rate, an AI recognition model configuredto recognize product information by receiving the video of the product,an artificial neural network processor configured to drive an AIrecognition model at a second frame rate, and a display moduleconfigured to display a video of a product at a first frame rate and todisplay product information at a second frame rate.

The first frame rate and the second frame rate may be the same. Thefirst frame rate may be faster than the second frame rate.

A mobile artificial neural network device further includes a battery,and the camera or artificial neural network processor may be configuredto lower the first frame rate when the remaining charge of the batteryfalls below the first threshold value.

The first frame rate may be configured to be selectively adjusted inconsideration of power consumption of the mobile artificial neuralnetwork device.

The artificial neural network processor may be configured to include anoperation structure capable of performing an artificial neural networkoperation of an AI recognition model.

The mobile artificial neural network device may display an augmentedreality in a display module by superimposing product information on avideo of the product.

The mobile artificial neural network device further includes acommunication module, and the communication module may be configured totransmit information on the product to the server and to receiveadditional information on the product searched from the server.

The mobile artificial neural network device may transmit only productinformation among the product video and the product information to theserver through the communication module.

The mobile artificial neural network device may transmit productinformation to the server and may receive additional product informationfrom the server.

The AI recognition model recognizes consecutive images of a video of aproduct input from various angles, and when information of differentproducts among product information is recognized, the information ofdifferent products may be combined.

The information accumulated by the AI recognition model may be at leastone of product shape, color, trademark, name, manufacturer, and barcode.

The AI recognition model may recognize a video of a product and mayoutput information of at least one product in the order of a highrecognition rate.

The AI recognition model may be updated with the newly trained AIrecognition model through the server.

The AI recognition model may further include a plurality of mutuallydifferent AI recognition models.

The AI recognition model may recognize the GS1 standard productidentification code or barcode and may receive additional information ofthe product corresponding to the GS1 standard product identificationcode or barcode through the server.

The additional information on the product may include information on thelowest price corresponding to the information on the product.

The lightened AI recognition model may be applied with at least onelightening technique among pruning, quantization, model compression,knowledge distillation, and retraining, and AI-based lightening modeloptimization techniques.

The processor may be an artificial neural network processor, which is anNPU.

Features, structures, effects, and the like described in the embodimentsabove are included in one embodiment of the present disclosure, and arenot necessarily limited to only one embodiment. Furthermore, thefeatures, structures, effects, and the like illustrated in eachembodiment may be combined or modified for other embodiments by a personhaving ordinary knowledge in the art to which the embodiments belong.Accordingly, contents related to such combinations and modificationsshould be construed as being included in the scope of the presentdisclosure.

In addition, although the above has been described with reference to theembodiment, this is only an example and does not limit the presentdisclosure and those ordinary skilled person in the art to which thepresent disclosure pertains will appreciate that various modificationsand applications not illustrated above are possible without departingfrom the essential characteristics of the present embodiment. Forexample, each element specifically shown in the embodiment can bemodified and implemented. Thus, differences related to thesemodifications and applications should be construed as being included inthe scope of the present disclosure defined in the appended claims.

[National R&D project that supported this invention]

[Task identification number] 1711117015

[Task number] 2020-0-01297-001

[Ministry Name] Ministry of Science and Technology Information andCommunication

[Name of project management (professional) institution] Information andCommunication Planning and Evaluation Agency

[Research project name] Next-generation intelligent semiconductortechnology development (design) (R&D)

[Research Title] Advanced Data Reuse Development of Deep LearningProcessor Technology for Ultra-low Power Edge

[Contribution rate] 1/1

[Name of project execution organization] DEEPX CO., LTD.

[Research Period] 2020 Apr. 1˜2020 Dec. 31

What is claimed is:
 1. A mobile artificial neural network deviceincluding: a camera configured to output a video of a product at a firstframe rate; an artificial intelligence (AI) recognition model configuredto recognize product information by receiving the video of the product;an artificial neural network processor configured to drive the AIrecognition model at a second frame rate; and a display moduleconfigured to display the video of the product at the first frame rateand to display the product information at the second frame rate.
 2. Themobile artificial neural network device according to claim 1, whereinthe first frame rate and the second frame rate are the same.
 3. Themobile artificial neural network device according to claim 1, whereinthe first frame rate is faster than the second frame rate.
 4. The mobileartificial neural network device according to claim 1, furthercomprising a battery, wherein the camera or the artificial neuralnetwork processor may be configured to lower the first frame rate when aremaining charge of the battery falls below a first threshold value. 5.The mobile artificial neural network device according to claim 1,wherein the first frame rate is configured to be selectively adjusted inconsideration of power consumption of the mobile artificial neuralnetwork device.
 6. The mobile artificial neural network device accordingto claim 1, wherein the artificial neural network processor isconfigured to include an operation structure capable of performing anartificial neural network operation of the AI recognition model.
 7. Themobile artificial neural network device according to claim 1, whereinthe product information is superimposed on the video of the product todisplay augmented reality in the display module.
 8. The mobileartificial neural network device according to claim 1, furthercomprising a communication module, wherein the communication module isconfigured to transmit information on the product to a server and toreceive additional information on the product searched from the server.9. The mobile artificial neural network device according to claim 8,wherein the mobile artificial neural network device is configured totransmit only product information among the video of the product and theproduct information to the server through the communication module. 10.The mobile artificial neural network device according to claim 8,wherein the mobile artificial neural network device is configured totransmit product information to the server and to receive additionalproduct information from the server.
 11. The mobile artificial neuralnetwork device according to claim 1, wherein the AI recognition model isconfigured to recognize consecutive images of the video of the productinput from various angles, and wherein, when information of differentproducts among the product information is recognized, the information ofdifferent products is combined.
 12. The mobile artificial neural networkdevice according to claim 11, wherein the information accumulated by theAI recognition model may be at least one of product shape, color,trademark, name, manufacturer, and barcode.
 13. The mobile artificialneural network device according to claim 1, wherein the AI recognitionmodel is configured to recognize the video of the product and to outputinformation of at least one product in the order of a high recognitionrate.
 14. The mobile artificial neural network device according to claim8, wherein the AI recognition model is configured to be updated with anewly trained AI recognition model through the server.
 15. The mobileartificial neural network device according to claim 8, wherein the AIrecognition model is configured to include a plurality of mutuallydifferent AI recognition models.
 16. The mobile artificial neuralnetwork device according to claim 8, wherein the AI recognition model isfurther configured to recognize a GS1 standard product identificationcode or a barcode and to receive the additional information of theproduct corresponding to the GS1 standard product identification code orthe barcode through the server.
 17. The mobile artificial neural networkdevice according to claim 8, wherein the additional information on theproduct is configured to include information on the lowest pricecorresponding to the information on the product.
 18. The mobileartificial neural network device according to claim 1, wherein the AIrecognition model is a lightened AI recognition model.
 19. The mobileartificial neural network device according to claim 18, wherein thelightened AI recognition model is applied with at least one lighteningtechnique among pruning, quantization, model compression, knowledgedistillation, and retraining.
 20. The mobile artificial neural networkdevice according to claim 1, wherein the processor is an artificialneural network processor, which is an NPU.